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§0. Introduction 

Let X denote a curve of genus g over a finite field Fg. By the Riemann hypothesis for 
function fields, 

|X(F,)| = 1-T + Q, 

where 

(0.0.1) ~2g^<T<2g^. 

We consider this inequality from a naive probabilistic point of view. Suppose g = 1, for 
example. Then E has a model as a projective cubic curve. Let F{x, y, z) denote the 
corresponding homogeneous cubic polynomial. There are q'^ + q + 1 F^-rays through the 
origin in affine 3-space, and loosely speaking, each ray has a probability 1/q of lying on the 
zero-locus of F. Therefore, X{¥q) should should have about q + 1 points, with an expected 
error of O(y^). Thus (0.0.1) gives the right order of magnitude. On the other hand, 
rather than satisfying a Gaussian distribution law as one might expect, T/ ^Jq is absolutely 
bounded by 2. More precisely, in the limit q — > cxd, the values of Tj ^ are uniformly 
distributed with respect to a certain measure /ii, known as the Sato-Tate measure, if E 
is drawn at random from the set of isomorphism classes of elliptic curves over Fg. For 
each value of (7, we obtain in this way a measure iig supported on [—2g,2g]. The main 

result of this paper is that the limit of these measures is, in fact, ^ = (27r)~^/^e~^ dx. 
Heuristically speaking, for a random curve of random genus, over a random finite field, 
T I ^fq is normally distributed. 

The distribution of |X(Fq)|, as X varies over a family of varieties, can be approached 
via the cohomological theory of exponential sums, due to Deligne [5]. More generally, to 
a family of exponential sums (suitably defined) one can associate a compact Lie group 
G, the geometric monodromy group, and a finite-dimensional representation (p, V) of G 
with character x- Under suitable hypotheses, the distribution of values of the sums is the 
same as the distibution of values of x((7), as g is drawn randomly from the uniform (Haar) 
measure on G. For each genus g > 2, we construct such a group Gg and a symplectic 
representation Vg. It turns out that Gg is the full (compact) symplectic group Sp{2g) and 
Vg is its standard 2(7-dimensional representation. It is not easy to explicitly compute the 
distribution of xio) as g ranges over Sp{2g), but the moments of the distribution can be 
read off from the invariant theory of Sp(2(7). There is a precise sense in which the invariant 
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theory "stabilizes" for large values of g. The measure-theoretic counterpart of this fact is 
the statement that fig converges to ^. 

Of course, the heuristic principle suggesting this theorem is not limited to curves. 
One would also expect, for example, that a "random" hypersurface in of degree k ^ 
should have + g + 1 + T points, where T/q is normally distributed. The corresponding 
Vk in this case is orthogonal. Fortunately, the invariant theory of SO(n) also stabilizes 
(though one should be careful to distinguish even and odd values of n), and the limit of the 
Sato- Tate measures associated to the standard representations is again fi. To prove that 
the Sato- Tate measures of {Gk,Vk) for hypersurfaces of degree k do indeed tend /x, one 
would need to show somehow that Gk is "as large as possible." In most known instances, 
geometric monodromy groups associated with self-dual representations do turn out to be 
either Sp, O, or SO. See, for instance, [8] 11.1 and [9] 8.13 for a discussion of Kloosterman 
sums and hypergeometric sums respectively. 

This paper is organized as follows. The first section analyzes the geometric mon- 
odromy group (G'2g_|_2, ^23+2) of ^ subfamily of the curves of genus g, namely, the hyper- 
elliptic curves of degree 2g + 2. We compute the first 2g + 1 moments of the associated 
Sato-Tate measure 1^23+2 explicitly. These turn out to be the same as the first 2g + 1 
moments of fi. To conclude that fi is actually the weak limit of the fi'2g+2 requires some 
work. The second section gives an effective version of the Weierstrass approximation the- 
ory which justifies this conclusion. At this stage, we still do not know that ^2^+2 is Sp2g, 
though the values of the moments arc highly suggestive. The third section is pure invari- 
ant theory, devoted to an analysis of the fourth moment of the Sato-Tate measure of a 
self-dual representation of an infinite compact group. Its value is 3 when {G, V) consists of 
the standard representation of a symplectic, orthogonal, or special orthogonal group, and 
with one exception, the converse is also true. With this result in hand, we return to ^2^+2 
in the fourth section and prove that is exactly Sp2g. Since the full moduli space of curves, 
Aig, is larger than the family of hyperelliptic curves, we expect the geometric monodromy 
group Gg for to be at least at large as ^23+2- We make this precise, and prove that 
in fact Gg = ^23+2- 

The results of this paper are superficially similar to those of Davidoff [4]. However, the 
apparent similarity is illusory. Let G^ = U(l)'^ with its standard n-dimensional represen- 
tation. If the associated Sato-Tate measures are rescaled by a factor of ^/n, by the central 
limit theorem, they will converge to the normal distribution. This is a completely different 
phenomenon from the (rather miraculous) fact that the Sato-Tate measures of the stan- 
dard representations of orthogonal and symplectic groups, without any renormalization, 
tend to ji. Davidoff considers a special family of hyperelliptic curves which are endowed 
with many endomorphisms, and as a result, she obtains a commutative monodromy group. 

This paper has benefited from discussions with C. Epstein, N. Katz, A. Kouvidakis, 
G. Kuperberg, A. Lindenstrauss, and R. Pink. It gives me pleasure to acknowledge their 
assistance. 

§1. Hyperelliptic Exponential Sums 

In this section we compute the k^^ moment of the generalized Sato-Tate measure of 
of the universal hyperelliptic curve of genus ^ for A; < 2^ -|- 2. These measures behave 
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as one would expect if the geometric monodromy group of the sheaf of cohomology groups 
were Sp{2g). This cohomological point of view is explained below. We begin with some 
estimates of exponential sums. 

(1.1) Let p denote a fixed odd prime and n > 3 an integer. Let Rn/¥p denote the affine 
variety 



Rn = Spec ( E 



^1 ) • • • ) ^nj 
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tl — t2 ' ^1 — ^3 ' ' tn-1 

of ordered n-tuples with no repeated elements. We consider the universal hyperelliptic 
curve X over Rn given by the homogeneous equation 



(1.1.1) 



Y[{x-Uz) = 0. 



The unique point at oo on this curve has projective coordinates (0, 1,0). Let Fg denote a 
finite extension of Fp and x the extension by zero of the quadratic character on F^ For 
each n-tuple a = (ai, . . . , a^) e F^ , and each a; e Fg, the cardinality of the set of solutions 
y e Fg of 

y'^ = {x-ai)---{x- an) 

is 1 + — (ii) ■ • ■ {x — an))- Summing over x, we see that the cardinality of Xg,{¥q) is 
g — T + 1, where 



(1.1.2) 



Ta = - ^ X{{x -ai)---{x- an)). 



Lemma (1.2) If P{x) a polynomial over a finite field Wq, then 



a;€F, 

unless P{x) = cQ{x)^ for some Q{x) G Fq[a;], c e Fg. 
Proof. If P{x) = R{x)Q{x)^, then 

I J2 xiPix)) - E xiRix)) \< deg(Q). 

Therefore, without loss of generality, we may assume that P{x) is squarefree. Let d denote 
the degree of P. If d = 0, there is nothing to prove. If d = 1 or d = 2, the sum is always 
or ±1 respectively. Otherwise y'^z'^~'^ = z'^P{x/ z) defines a projective hyperelliptic curve 
X of genus g — [^^] > 0. By the Lefschetz trace formula, 

\X{¥q)\ = q-tv{Fq:H\X,Qi)) + l, 

where Fq denotes the Frobenius element x ^ x'^, and H^{X, Qg) the etale cohomology of 
X with £-adic coefficients, I ^ p. By (1.1.2), — XlxeF ^he trace of Frobenius on 

. By the Riemann hypothesis for curves over finite fields, this has absolute value less 
than or equal to 2g^. 
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Lemma (1.3) Let P{x) he a monic polynomial over a finite field Wg, not a perfect square. 
Then 

{ai,...,an)eRn (Fq) 

Proof. For each partition tt of a set S, we write Ftt.e for the subvariety of the 
dimensional affine space defined by the equations Xa = xi, whenever {a, 6} C k e tt. 
Setting = {1, 2, . . . , n}, we consider 

S.,^^= x{P{ai)P{a2)---P{ar,)) 

(ai,...,a„)eK,E„(F,) 

and prove by induction on n that all of such sums are 0(g'^/^). If k e tt is a singleton {k}, 
then 

'5.,E„= I Yl x(^(«'^)))'5AM,E„\{fe} = C»(9"/') 

by Lemma (1.2) and induction on n. If all the equivalence classes of tt have two elements 
or more, there can be no more than n/2 of them, so dim(V^ e ) < n/2. It follows that 

By the inclusion-exclusion principle, the characteristic function on is a linear 
combination of the characteristic functions of e • The lemma follows. 



Proposition (1.4) Let 

F{m) = 
For m < n, 



(m — 1)!! := (m — l)(m — 3)(m — 5) • • • (5)(3)(1) if m is even, 
if m is odd. 



Proof. By (1.1.2), 

(n m \ 
nn(-.-«-) • 
i=ij=i J 

Defining 

m m 

i=i j=i 

we obtain 
(1.4.1) 

E (-^a)"^- E ••• E E x{{~irnx{p.,,...,.M---p.,,...,.M)- 



If there exists a partition tt of into pairs such that (xi, . . . ,Xm) £ then 
as a polynomial in y, Pxi,...,x^{y) is a perfect square, but otherwise it is not. Therefore, 
outside the union of such loci, the inner sum of (1.4.1) is of order 0{q^^'^). There are F{m) 
partitions tt of into pairs, and the loci Vt^ intersect pairwise in sets of lower dimension, 
so 

aefl„(F<,) 

Remark (1.5) If m and n are both odd, it is easy to see that 
Indeed, if A e is chosen such that x(-^) = — 15 then 

TAa = - X{{x - Aai) ■■■{x- \an)) = - J] x(A^)x((a^ -a{)---{x- an)) = -Tx^- 

(1.6) Let S/¥p denote a geometrically connected variety and s a fixed geometric point 
of 5". The functor assigning to each finite etale X/S the set Hom5(s, X), is represented by 
a projective system {Xi, (f)ij : Xj — > X^} of finite etale covers of 5'. As usual, we define 

7rt^{S,s) :=iimAutx(X,). 

i 

Given a second geometric point s', there is an isomorphism 7r^^^(5', s') = 7r^^^(5', s) canonical 
up to inner automorphism {i.e., up to choice of "path") [11] I 5.1 (a). Let S denote the 
variety obtained from S by extension of scalars to Fp. There is a short exact sequence 

^ 7rf^{S, s) ^ 7rf^{S, s) ^ Z ^ 

([5] 1.1.13). By a smooth £-adic sheaf on S, we mean a continuous homomorphism 
TT^ ^{S, s) —>■ GL^(Q^). We fix an embedding l : Qi —>■ C and say that J-' is pure of weight 
e Z if for every Fg-point x e S{¥q), every geometric point x of x, and every choice of 
path Trf ^(5, x) Trf ^(5, s), the image of Frobenius under the composition of arrows 

Trf^ix, x) ^ Trf^iS, x) ^ 7rf^{S, s) ^ GL,(Q,) ^ GL„(C) 

has all its eigenvalues of absolute value g"'/^. We write tr(^a;) for the trace of the resulting 
matrix. 
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(1.7) Let d = dim(5). By Poincare duality ([11] VI 11.1) 

The right hand side is pure of weight w + 2d. Let Gseom Jenote the Zariski closure of 
7rf^^{S,s) in GL„(Q^). The space of Gs®°™-invariants in Q" is the same as the space of 
7rf^{S, s)-invariants, so 

dim(iy,2^(^,.F)) = dim(.F«^^°""). 

Note that by [5] 1.3.9, 3.4.1 (iii), the identity component of Gseom semisimple. By the 
Lefschetz trace formula [11] VI 13.4, 

2d 

J2 tr(.F,) = J](-l)MF,:iy^(5,.F)). 

a;eS(F<,) i=0 

By [5] 3.3.1, the i = 2d term dominates the sum for g ^ 0. Therefore, if 

^^^^n(-d-W2) J2 tr(.F,) 

x£S{Fpn) 

tends to a limit as n — > oo, then H^'^{S, T) must be just Qi{—d — w/2), and 
(1.7.1) hm a, = dim(.F«'^''°""). 

n— >oo 

Proposition (1.8) Let X-^R^ denote the universal hyperelliptic curve over the base Rn, 
and T = i?^7r*(Q£) the sheaf of of the fibres over R^,. Then T is pure of weight 1 and 
for m < n, 

dim((jr®^)^''°") =F(m), 
where F{m) is defined as in Prop. (1.4)- 

Proof. The purity statement is immediate from [5] 3.4.11. This implies that JF®'" is pure 
of weight m for all positive integers m. The proposition now follows immediately from 
Prop. (1.4) and (1.7.1). 

(1.9) Let G denote an algebraic group over C with semisimple identity component and 
(p, V) a complex representation of G. There is a unique compact real form G*^ of G, and 

dim(F^) = dim(F^'). 

On the other hand, if /j, denotes Haar measure on G^, then 

dim(y^') = / Xp{9)l^i Xp ■= trop. 
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We define the generalized Sato-Tate measure for {G^,V) as the direct image measure 

f^s-T ■■= Xp^f^ 

on M. In particular, we can associate a Sato-Tate measure to every pure sheaf over S. By- 
definition of direct image, 

(1.9.1) rx^fxs-T=[ Xp{9rf^= f Xp«"(^)/^ = dim(K®-^). 

Prop. (1.8) can be interpreted as the computation of the first n — 1 moments of the 
Sato-Tate measure associated with the family of hyperelliptic curves over 

§2. Moments and Limits of Non-negative Measures 

(2.1) Let / : R — > R'*^ denote a positive real valued-function. We say / is very rapidly 
decreasing (VRD) if every smooth compactly supported function ^ is a uniform limit (on R) 
of functions of the form f{x)P{x), where P{x) is a polynomial. 

Lemma (2.2) If f{x) is VRD, then so is f{Xx) for all x. 

Proof. The ring of polynomials and the L°° norm are both invariant under the rescaling 
X h->- Xx. 

Lemma (2.3) If f{x) is VRD and g{x) is positive, bounded continuous function, then 
f{x)g{x) is VRD. 

Proof. Approximating a smooth compactly supported function h{x) by f{x)g{x)P{x) in 
the L°° topology is equivalent to approximating h{x)/g{x) by f{x)P{x). 

Lemma (2.4) If f{x) is VRD, then the moment integrals 

/oo 
f{x)x''dx 
-oo 

converge for all n. 

Proof. The space of smooth compactly supported functions on R is of infinite dimension, 
and all such spaces are uniform limits of sequences f{x)Pn{x). Therefore, f{x)P{x) G 
L°°(R) for some polynomial P of arbitrarily large degree. Therefore f{x) — o{x~^) for all 
n, and the lemma follows. 
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Proposition (2.5) The normal functions f(x) = e are VRD. 

Proof. Without loss of generality we may assume K = 1/2. Let Tn{x) denote the n^^ 
Chebyshev polynomial of the first kind. When x e [—1,1] and 4|n, we have 



For |a;| < 1 < ^, 



so when 4|n, 



Tnix) — cos(ncos ^ x) — cos(nsin ^ x). 



™2 ^2 

1 > cos a; > 1 , — < sm X < a; , 



1 - 2r?x^ <Tn{x)<\- '—x^ 

for I a; I < Let m denote an integer congruent to 2 (mod 4). For \x\ < m/2, 

2x^ , ,n/a;\2 (m — 2)^/x\2 

(2-5.1) < l-2(™-2)^ (^) < < 1-^-^ (_) < 1- 



X 



m 

We define 



16m2 



1 4 ) 



As (1 - e^/' > 1/4 for < e < i, for |a;| < 



m 



-1/2 



Qr 



,{x) > ((1 - m-^)(l - 2m-3))"^'/4 > (1 _ 4to-3)^V4 > }_ 



As m^/4 is even, Qm{x) > for all x, so 
In the other direction, (2.5.1) implies 



1 



2^/m 



(2.5.2) 



/ 2 \ 



IQm? J 

for \x\ < m/2. For m/2 < |x| < ■\/2m^ — m^/4. 



(2 \ 



(2.5.3) 



Finally, we note that the identity T„(cosha;) = coshnx implies that for \y\ > 1, 

Tniy) < \2yr. 
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For |a;| > m^, then, 

< gm(a;)e-^^-'')'/2 ^ (2"^^-2'"a;'^)'^Vv(x-r)V2_ 

From the fact that \x'^e~^ /^l achieves its maximum when x — ±\/n, we deduce 
sup a;^e-(^-^)'/2 = sup(a; + r)"e-^'/2 = sup(l + r/a;)^e-^'/^ 

x€R xeR x€R 

If the maximum is achieved for some x >2r, then 

supx'^e-^^-'')'/^ < n^/2(^-"/^ 

a;elR 



where C = 4e/9 > 1. Thus, if r < 



9C' 



SUpX^e-^^-^)'/^ < s^p( ^ng-(x-r)V2^^n/2(^-n/2) < ^n/2(<.-n/2_ 

xeK |a;|<2r 



2 

We conclude that if r < 



6^/C' 

(2.5.4) Q^(a;)e-(^-^)'/2 < (2^m-2"^)^'/^ supx^'/^e-^^-^)'/^ < C"^. 



For each n e N, we set m = 4n + 2 and define fnix) as the product of Qm/Im and the 
characteristic function of the interval [—m'^,m,'^]. Then fn is a non-negative measure of 
integral 1, and the estimates (2.5.2) and (2.5.3) imply that is an approximate identity. 
Therefore, if (7 is a smooth compactly supported function, the sequence of convolutions 
fn* 9 converges uniformly to g. By (2.5.3) and (2.5.4), since g is supported on a subset of 
[— r, r] for some r, 

e~'''/^((/n - Q4n+2/hn+2) * 5^) 

tends to zero uniformly on R as n — > oo. Therefore, g is the uniform limit of the products 
g-a; /2((Q4„+2/-f4n+2) * ^) of a fixed Gaussian with a sequence of polynomials. 

Proposition (2.6) Let //i,//2, • • • denote a sequence of non-negative measures on the real 
line and f{x) a smooth positive real-valued VRD function. Suppose that for all m < n 

x'^fXn= / X'^fixfdx. 

-OO J — OO 

Then converges to f{x)^ dx in the weak-* topology. 

Proof. We note that f{xY is VRD, so the right hand side of (2.6.1) converges. Let 
g{x) by any smooth compactly supported function which takes only non-negative values. 
Choose e > 0, and let h{x) denote a smooth compactly supported function such that 
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h{x) = -y/ g{x) + e on the support of g and — g{x)\oo = e. Now f{x)h{x) is the 

uniform hmit of a sequence f{x)An{x), where G M\x\. For n sufficiently large 

{f{x)Ar,{x)f > f{xfg{x) 

on M. Letting e tend to 0, we can write f{x)'^g{x) as a uniform limit of functions 
f{xfB^{x), where G M[x], and B^{x) > g{x) for all x E R. As /(a;)^ is VRD, 

it is integrable, so f{x)'^Bn{x) converges in the norm to f{xYg{x). Therefore 
(2.6.2) 

g{x)nn < lim lim / Bm{x)nn = lim / Bm{x)iJ, = / g{x)iJ. 

-OO Tn—^oo'"'~^°° J -OO m^ooJ-oo J-oo 

On the other hand, the measure of the real line is the same with respect to and with 
respect to fi^ for n ^ 0, so we must have equality in (2.6.2). Finally, every smooth 
compactly supported function is the difference of two such functions which are everywhere 
non-negative, and the proposition follows. 

Corollary (2.7) If n'^ denotes the Sato-Tate measure associated with the family of hyper- 
elliptic curves over Rn, then in the weak-* topology, 

(2.7.1) lim = -^e-^^/^dx. 



/. 



Proof. By Prop. (1.8) and (1.9.1), 

/■OO 

for m < n. Applying Prop. (2.6) to (27r)~-'^/^e~^^/^, we deduce (2.7.1) from the integral 

x^e-^^/^dx = V2^F{m), 

' — OO 

which is easily checked by integration by parts. 

§3. A Problem in Invariant Theory 

Throughout this section G will always denote a compact Lie group and V a faithful 
finite dimensional representation. 

(3.1) Given a fixed G and V, we define the sequence of Sato-Tate moments 

an = an{G, V) = dim j^V®"^) , n > 1. 

It is possible, in general, that ai = a2 = ■ ■ ■ = dm = 0, for any desired value of m. This is 
the case, for instance, if G = SU(n), V is the standard n-dimensional representation, and 
n> m. However, if V is self-dual, then a2k > for all k. Indeed, 

a2k = dim (^F®2fcG^ ^ ^.^ ^^^^^ ^y^k^y^k^^^ ^ Y^^2^ 

i 

where rrii denote the multiplicities of the irreducible factors of V®^. In particular, a2k 
is at least as large as the number of summands appearing when V®*' is decomposed into 
irreducible representations. 

Henceforth, we shall always assume V is self- dual. 
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(3.2) This section is devoted to the classification of pairs (G, V) such that 04 (G, V) = 3. 

The complete classification problem seems to be quite difficult. A wide variety of interest- 
ing finite groups admit such representations. For example, the irreducible 3-dimensional 
representations of satisfy this condition. So do the standard 6, 7, and 8 dimensional rep- 
resentations of the Weyl groups of £^65 Er, and Eg respectively. The natural 24-dimensional 
representation of the automorphism group of the Leech lattice does as well. The 133- 
dimensional representation of the Harada-Norton group and the 248-dimensional repre- 
sentation of the Thompson group provide even more exotic examples. The equality 04 = 3 
can be readily checked in all these cases with the aid of character tables, such as those in 
[2]- 

Fortunately, in our application, finite groups may be ruled out on geometric grounds, 
so we are not obliged to attempt to classify the solutions. It seems likely that a complete 
list is attainable, using the classification of finite simple groups and the following lemma: 

Lemma (3.3) Let (G, V) denote a solution to the equation a4{G, V) = 3. Then every for 
every non-abelian normal subgroup H of G the restriction of V to H is irreducible. In 
particular, the centralizer of H in G has order < 2. 

Proof. First we observe that V must be an irreducible G-module. Indeed, ifV = V (BV" , 
then the trivial representation appears with multiplicity > 2 in Therefore, 04 > 4, 

contrary to hypothesis. Let H he a normal subgroup of G. As a representation of H, V 
decomposes into a direct sum 

(W^i © • • • Wk) ® C\ dim(W^i) = • • • = dim(W^fc) = m, 

where H acts irreducibly on the Wj and trivially on ([3] 49.7). As H is non-abelian and 
F is a faithful iJ-module, m > 1. Every element of G maps an iif-isotypic factor Wj 
into another such factor, Wj (g) C^. Therefore, 

k k 

V<^V = ^Sjm\W,^C^) © ^A\Wi^C^) © 0(W^i (g) C^) © (W^^ © 

i=l i=l i^j 

represents V ® V as a direct sum of three G-modules. If /c > 1 or £ > 1, all three pieces 
are have dimension > 1. Since there is also at least one G-invariant in F © F, 04 > 4. 
Therefore, k = £ = 1, and V is an irreducible ilf- module. By Schur's lemma, the centralizer 
of p{H) in GL(V) consists of the scalar matrices. But all traces of elements in p(G) are 
real, so only scalar matrices ±1 are possible. As V is faithful on G, the centralizer of H 
has order < 2. 

Corollary (3.4) If a4{G,V) = 3, then the identity component of G is either a torus or a 
semisimple group. 

Proof. Let G° denote the identity component of G. It is a normal subgroup of G. If it 
is not a torus, then it has a finite centralizer in G, hence a finite center. Therefore, it is 
semisimple. 
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Proposition (3.5) If a4{G, V) — 3 for an infinite compact group G, then G C GLiV) is 
Ns\j{2)U{\) C U{2), SO{n) C U{n), 0{n) C U{n), or Sp(2n) C U{2n). 

Proof. Suppose first that G° is a torus T. Then V is a direct sum Xi © • • • Xn of characters 
of T, and for each Xi there exists Xj = Xi^ ■ ^ irreducible as a G-representation, all 
the characters Xi must lie in a single orbit under the action of G/T on the character group 
X*{T). If some Xi were trivial, then all Xi would be trivial, contrary to the assumption 
that V is faithful. It follows that (suitably renumbering the indices), V is the direct sum 

xi ® xr' ® X2 e • • • © x^/V 

Therefore, as G-module, V decomposes into the following three pieces: 

(Xi ® Xi ® X»~' ® X^~^) ® (Xi ® X»~' ® X^~^ «) Xi) ® (Xi «) x7^ ® X»~' ® Xj-) • 

i i ij^j 

If n/2 > 1, then these pieces are all of dimension > 2. Since also has a G-invariant 
line, this implies 04 > 4. We conclude that n = 2, so G is contained in SU(2). As T is 
normal in G, G can only be the normalizer of a maximal torus. 

Suppose, on the contrary, that G° is a semisimple group. As G° is normal in G, the 
restriction of an irreducible representation of G to G° is the direct sum of highest weight 
modules Vx^ of G° , where the A j lie in the same orbit of the automorphism group F of the 
root system $ of G°. We apply this observation to V^^. The dual of the Killing form 
gives a F-invariant inner product on the space of characters of G, so it suflBces to find 
submodulcs V^^ of V^'^ of four different lengths. 

Given a semisimple Lie algebra q and a representation V, let 5(0, V) denote the set 
of norms ||A|p, where Vx is a submodule of V. If (f), W) is a second pair, then 

SiQxi),V^W) = {x + y\xe SiQ,V), y e 5(f), W^)}, 

so 

\SiQ xl),V^W)\> |5(0, V)\ + |5([), VF)| - 1. 

If is simple and Vx is faithful and self-dual, then 15(0, Vx) \ > 2 because 0, 2A G 5(0, V^)- 
By [10], lAf^ contains a submodule in every Weyl orbit W/j, — W{X + wA), for fixed 
w e W. If rk(0) > 1, this implies that |5(0, Va) > 3 because W acts irreducibly on the 
root space, and therefore some wX {±A}. Since V is self dual, WX is invariant under 
multiplication by —1. Suppose there exists w such that 

wX ^ {A,-A}UA^. 

Then A + A, A + wX, X — wX, and A — A are all of different lengths, so |5(0, Vx)\ > 4. 

We have seen that if = Lie(G°), and is a representation of G such that a4(G, V) = 
3, then |5(0, y)| < 3. In view of the foregoing analysis, this implies that q — 5I2 x SI2 
with V the exterior tensor product of the two standard representations, or is simple 
with V = Vx, where jJ-i, fi2 £ WX implies /ii = ±//2 or fii ± fi2- By the classification 
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of simple Lie algebras, the latter condition implies that g is of type Ai, (n > 2), C„ 
{n > 2), or Dn {n > 3), and A is a positive integral multiple of the fundamental weight 
Ui = (1, 0, . . . , 0), in the notation of [1] VI Planches. Note that the Lie algebras B2 and 
C2 are the same, but the value of ui depends on which name we choose. 

Assume that n = rk(g) > 1. We recall the Freudenthal formula for the multiplicity 
(A) of a weight A appearing in the highest weight module K; ([1] VIII §9 Ex. 5 (g)): 



{lo + p,u} + p) - {X + p, X + p) 



where p denotes the half sum of roots. Applying this formula for Cn, and to oj = 
koji and A = (fc — 1, 1, 0, . . . , 0), in each case we get a multiplicity of 1. As (2fc — 1, 1, 0, . . . , 0) 
appears with multiplicity 2 in V^_^, V(2fc-i,i,...,o) is a submodule of V^®^. By [10], we know 
that there are also submodules V(2fe,o,...,o); ^(fe,fe,o,...,o); and Vq. If /c > 1, the highest weights 
of these modules are all of different lengths. We conclude that X = uji. As F is faithful, 
this means that G° is S0{2n + 1), the compact form of Sp{2ri), or S0{2n), depending on 
whether g is of type B^, Cn, or D^. In any case, G is contained in the normalizer of the 
image of G° under its standard representation, so G = G° in the symplectic case and G 
can be either of type 50 or type O in the orthogonal case. 

Finally we consider the sh cases. When g = sh, V = Vfec^^, then \S{q, = k+1. 

When k = 1, we have the standard representation of G° = SU(2), so 04(0, V) < 3. 
When /c = 2, we have the solutions S0(3) and 0(3) enumerated above. Finally, if = 
s/2 X s/2, the condition \S{g, V)\ < 3 implies that V is the exterior tensor product of the 
standard representations of the two factors. This gives rise to the solutions SO (4) and 
0(4) enumerated above. 

§4. Monodromy for the Moduli Spaces of Curves 

Theorem (4.1) For n > 5, the geometric monodromy (G^, Vn) of R^n^Qi for the family 
of hyperelliptic curves over Rn is Sp2[(n-i)/2]- 

Proof. Applying Prop. (1.4) for m = 4, we see that 04(0^, Vn) = 3. Now dim(F„) = 
2g = 2[(n — l)/2], so by Prop. (3.5), G'^ is finite, symplectic, or orthogonal. On the other 
hand, the cup product on is anti-symmetric, so by Poincare duality, is a symplectic 
representation. Therefore, it suffices to prove that G' is infinite. Now 



Rn = Spec (w 



tl, ■ ■ -jtn, , , 5 • ■ ■ 7 , , 

II — I2 II — I3 In-1 ~ In 



is an open subvaricty of affine space A"^ over F, and X extends naturally to the projective 
curve over tt : X ^ A"^ defined by (1.1.1). For every non-singular curve Z C A"^, we 
define Z° = Z C] R^- We choose Z such that Z \ Z° is a non-empty subset of the smooth 
locus of A"^ \ Rn- The restriction Xz :— X x^n Z is a Lefschetz pencil, and the restriction 
of R^7rz*Qe to Z° is a smooth £-adic sheaf with finite monodromy. Therefore, there 
exists a finite etale cover Y° of Z° on which is a sheaf with trivial monodromy. The 
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normalization of Y° over Z is a non-singular curve Y with a Lefschetz pencil Xy — > Y. 

The fibres over the (non-empty) set Y \ Y° have a double point, so there is at least one 
vanishing cycle of R^ny^Qt- By the Picard-Lefschetz theorem ([11] V 3.15), the space of 
TTi-invariants of -R^TTyo^Q^ is orthogonal to the space of vanishing cycles under the Poincare 
pairing. As the Poincare pairing is perfect, we have a contradiction, and the theorem holds. 

Remark (4.2) In the light of this result, the moment computations of Prop. (1.8) give 



a^(Sp(2^), Std) = { _ i)„ Is even. 



for m < 25r-|-l. This is a classical result of invariant theory [13] 6.1.A, 6.I.B. The analogous 
result also holds for orthogonal groups [13] 2. 11. A, 2. 17. A. 

(4.3) For each pair {g, n) of non-negative integers, consider the functor of n-pointed 

curves of genus g over Fp, i.e., the functor assigning to each scheme S/¥p the set of n -f- 1- 
tuples (tt, (Ti, . . . , (Tn), where tt : X ^ 5" is a proper smooth morphism and ai : S ^ X are 
sections of tt such that for all geometric points s on S, Xg is a curve of genus g and o'i{s) 
are distinct points. For fixed g, when n is sufficiently large, this functors is represented 
by a quasi-projective variety M.g,n [12] 11. By a theorem of Deligne and Mumford [6], 
the coarse moduli space of curves Mg and therefore the variety A^g,n is geometrically 
irreducible. Let l23+2,n denote the complement of the diagonal on the n-fold fibre power 
of the universal curve on i?2g+2- From the projection Y23+2,n ~^ -R23+2 we obtain a 
universal curve on l2g+2,n, with n canonical sections. The data of curve with sections 
defines a map ig,n ■ l2g+2,n ■^g,n- The universal curve on A4g,n (resp. Y2g+2,n) gives 
rise to a sheaf of relative first cohomology groups, and therefore to a continuous £-adic 
representation ttm of 7r^^^(7Vfg^„, m) (resp. TCy of 7r^^^(l2g+2,n, y)- If we choose m to be 
the image of a fixed geometric point y of l2g+2,n, we obtain the commutative diagram 

GUgim 

(4.4) The representation (tt^, Vy) is obtained by pull-back from the continuous represen- 
tation of Til ^{Rji, f^j) on the hyperelliptic curve Xfh- Therefore, the geometric monodromy 
group Gy of the sheaf of relative on l2g+2,n is a subgroup of Sp(25f). There are several 
ways to see that it is actually the full group. One way is to construct a multi-section 
Rn — > 5^2g+2,n- Another is to note that the inequality of exponential sums 

{q+l-2g^-nr Yl < E Tf < (g+l+2^^-n)" J] 

aefl„(Fq) (a 

implies 

J2 = F{2k)q''+^3+k + Q(^qn+2g+k-l/2^^^ 

{a,Xl,...,Xn)€Y2g+2,n0^q) 

and therefore 

dim(y®2fcGv.j ^ ^^2fc) = dim(y®2fcSp(2p)^_ 
It follows from a standard result in invariant theory [7] I Prop. 3.1 (c) that Gy — Sp{2g). 
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Theorem (4.5) Let p : X ^ ■^g,n denote the universal curve of genus g. Then the 
geometric monodromy Gg^n of R^p^Qi is Sp{2g). 

Proof. The map ig^n realizes the geometric monodromy group G2p+2,n 

universal curve on Y2g+2,n as a subgroup of Gg^n- On the other hand, by Poincare duality, 

Gg^n is contained in Sp{2g). The theorem follows. 

Remark (4.6) We do not state a monodromy result about the moduli space Mg itself 
because it does not admit a universal curve. There is an algebraic stack M.g, and presum- 
ably the natural language in which this theorem should be framed is that of smooth £-adic 
sheaves on stacks. The choice of scheme language was dictated by the lack of adequate 
references on the foundations of the theory of stacks. 
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